Talk:M File Format
MIDI Music vs FM Oh. My. God! Like... seriously! So I was futzing with my MIDI writer, and I noticed that sometimes I would get "No instrument mapped to tone bank 0" warnings. Further investigation led me to note that my sound font was extremely incomplete, and not the least bit out of date. So I updated it, and... just... WOW! I had no idea how POWERFUL the MIDI music could actually sound. 15 years ago, MIDI sound fonts were crap, that's why I was working on the FM player. Today, it sounds like there's an entire orchestra playing just for me! Seriously, I played BIGTHEME and was just blown away. The FM version of SF05.M has NOTHING on the MIDI version. --WizardStan 21:19, 28 June 2009 (UTC) M to Midi converter? I don't suppose you've worked on an M to MIDI converter? I took a quick shot at it, but the note values in the M files often exceed 127, which is the maximum allowed by MIDI. When I compared an M and MIDI (dumped from DOSBox) of the same song, it looked like the note values in the MIDI were roughly half the corresponding note values in the M, but it was never exactly one half, and occasionally, the values were equal. So, I'm completely stumped. -- 23:24, 24 June 2009 (UTC) No, I haven't dealt with the MIDI portion of the M files very much. A .M file isn't even a MIDI file. It's a custom music format that contains instrument definitions and then a string of commands, the commands not having a 1:1 relationship with any of the Midi commands themselves. The instrument definitions include FM, MIDI, PC Speaker, and possibly a few others that I hadn't bothered to look into. It plays the different instruments in different ways depending on which driver was loaded. Personally, I found the FM music to sound the best, so once I worked out how to play back the FM instruments, I didn't go much beyond that. If there's demand, I'll certainly look into how to parse the MIDI portion of the .M file and create a MIDI file from that. If someone wants to have a play around with what I hacked together, download the source at http://www.rebirthofxeen.com/files/musictest.tar.gz It uses SDL and the same OPL emulator in DosBox. It compiles in both Windows and Linux. -- WizardStan 00:31, 25 June 2009 (UTC) I realize they're not the same; were they, I wouldn't be here asking about a converter! :) What I did was parse an M file according to the specifications on the two related wiki pages here, parse a MIDI capture of the same song using a third party library, and compare the resulting dumps. The relationship between the two is actually very tight for a while. For instance, M instructions 0x2, 0x6, 0x9, and 0xA do, in fact, map 1:1 to MIDI events. Even most of the parameters are the same; only the note values are significantly different. It's not until M instructions 0x1 and 0x8 that the relationship really breaks down. Anyway, thanks for the code; I'll check it out. UPDATE: I compiled and ran the program but it doesn't make any noise. All I get are status updates, like these: TEMPLE.M Initializing SDL. SDL initialized correctly keyScale is 128 for instrument 0 on channel 0 keyScale is 128 for instrument 3 on channel 2 keyScale is 128 for instrument 3 on channel 4 keyScale is 128 for instrument 3 on channel 6 Hard quit requested. Stopping. I did notice in the code you did some extra parsing on the note value that isn't discussed on the wiki pages. It looks like you split the note value into two parts: The three most significant bits are the octave and the five least significant bits get mapped to some kind of frequency value (?). I don't really understand the frequency mapping. It looks somewhat similar to what's reported on the "Programming the...FM..." page linked off the M article, but the numbers aren't the same. Do you remember how the frequency part of the note value works? -- 15:04, 25 June 2009 (UTC) You know, this might be a little easier if you were to create an account. Or you could even email me: wizard (at) houseoffire (dot) ca :P Anywho, there's a bug in the code I posted. It's initializing the video system without creating a video window, which for some reason prevents the audio from playing. In musictest.cpp, the line: if(SDL_Init(SDL_INIT_VIDEO | SDL_INIT_TIMER | SDL_INIT_AUDIO)< 0) { should be if(SDL_Init(SDL_INIT_TIMER | SDL_INIT_AUDIO)< 0) { I'll have an updated download later tonight, but you should be able to make the change yourself. As far as the extra processing, I assume you mean the noteToFreq function. I forget where I grabbed the algorithm from, but it's a clever way of turning a midi note (with 0 being the lowest, 127 the highest, and middle C at 60) into its FM frequency equivalent. Each octave is divided into 256 possible frequencies, so it does some funky shift math to turn the given note into an octave/frequency byte pair, stored in a 16 bit integer. -- WizardStan 19:35, 25 June 2009 (UTC) Ok, I've started looking at the Midi engine. It actually looks really easy. There's some tweaking with the notes that needs to be done, which I'll experiment with this weekend, but otherwise it shouldn't be too much problem. I've already started some experimental stuff using PortMidi (an open source cross platform Midi API) and hope to have an update by Monday. -- WizardStan 05:23, 27 June 2009 (UTC) Cool. Now that I've gotten DOSBox up and running, though, I've discovered something horrible. All these years, I thought it was the MIDI versions I was hearing; that's why I was working on that MIDI converter. Turns out I was hearing the FM versions! While the MIDI versions do sound more realistic, I was surprised at how bombastic they are: They practically consume the entire gameplay experience! I much prefer the mellowness of the FM versions. I've run into a weird quirk with the M files. Some of them end in a big MIDI blob that lacks an 0xF7 terminator, and the M file itself lacks a pop instruction at the end. Your program refuses to even process them. This became an issue when I was trying to add some extra delay to the ends of songs for recording purposes. The songs with that quirk also tend to be the ones that most need a delay at the end, since they tend to loop immediately. Adding a delay and pop to the end of any M file with that quirk did not result in a delay (in the game). Maybe in your work with the MIDI player you'll figure out what's going on, there! -- 02:12, 28 June 2009 (UTC) Do you have an example? I've tried a handful of music files, and they all play just fine. Incidentally, I added MIDI playback anyway, with a preliminary MID file writer. The file writer mostly works, but seems to get corrupt about 10 seconds into any song for some reason. I can't figure out why yet. It uses PortMidi, a cross platform open source midi API. I'll try and have a Windows binary online Monday or Tuesday for those that have trouble compiling libraries. --WizardStan 04:21, 28 June 2009 (UTC) I can't remember which files were doing it, sorry. It triggered the 'WARNING: Reached end of file. File should end with cmd 0xF with \"channel\" 0xF to indicate reset, or any other value to stop playing.' message is all I remember. SDL_INIT_VIDEO was still in, I think. Also, I figured out that the quirk was actually my M parser's missing a few vital pieces. I compared my parser to your code and found a few places where I wasn't reading bytes and should have been (the wiki page claims there's no data bytes), and I wasn't moving around the M file at all in response to push and pop instructions (the wiki page isn't really clear on how to do it). It's fixed now, I think. At least, its dump of CASTLE.M makes a heck of a lot more sense, now. :) So basically, disregard the last paragraph of my last post. I think I better bow out now before I make a bigger ass of myself. :P -- 09:30, 28 June 2009 (UTC) Ah, the warning. Yeah, when I was first investigating, the files all ended in 0xFF to reset, or 0xF0 to indicate a stop playing. After I did the initial code, I threw some more music at it and a few would crash at the end, simply because they didn't have this terminating byte. So if it reaches the end of the file and doesn't have a terminator, it throws up the warning and assumes the file should be looped, because that's what the game does. I also have a tendency of figuring something out, coding it up, telling myself I need to fix the wiki, and then forgetting to do it. Sorry about that. I'll have a look and see if I can't clear up the file info. Alternatively, if you were to sign up, you could help contribute too, you know. :P --WizardStan 12:56, 28 June 2009 (UTC) I'm embroiled in a different revival project at the moment or I'd be more contributory. Plus, I don't seem to be very good at this at all. The project I'm on is a lot easier because I have the original C source code. (Which isn't to say it's easy. The C code is horrid--the second worst I've seen that wasn't intended to be bad. I'm not sure where someone gets the idea to #include 10 different .C files into a .H file, but I digress.) In other news, I did a little research about the Roland sound options. What it amounts to is that there's basically two different MIDI versions of each song: an MT-32 version and a Canvas version. The former is poorly emulated (IMO), and is the version I was basing my "I hate the MIDI versions of these songs" rant on. The Canvas version is pretty good, though. This isn't terribly surprising, since the MT-32 isn't a general MIDI synth, but the Canvas is. Anyway, I'm not sure if that will affect your work on the MIDI converter/player, but it's something to keep in mind. -- 05:14, 2 July 2009 (UTC) Good find with the Canvas/MT-32 thing. I was actually using what I think was the WaveBlaster driver (the filename is WAVEMUS) as the basis for my hacking, but intrigued by your note I investigate what I think is the MT-32 (driver filename ROLMUS) and the Canvas (CANMUS). Interestingly, they're all very close, but CANMUS had a few differences: most notably, it uses a different instrument set (Canvas references byte 25 of the instrument definitions while the other two use byte 24. FM uses the first 11 bytes to define its instruments, and the interceding 14 bytes are for PC speaker and the Pro-Audio Spectrum, supposedly) The Canvas driver also implemented another command in the M file which I had previously labeled as "skips two" because that's all it seemed to do: skip two bytes. Now I see it's used as a bank switch command to give access to more than just the basic 128 instruments. Without an actual Canvas to test against, the bank switching does nothing, unfortunately. Or maybe Windows is using some extended Midi magic and it is working, I can't tell. I'm sure it would sound a lot better if it could actually access the instruments it was intended for, rather than just the basic ones at any rate. As it is, some songs do actually sound better with the Canvas driver, but others sound better with the basic midi. You'll have to have a listen yourself. Which brings me to my next point: xeenmusic.exe is the most recent update to my music player to date. Given a .M file as an argument, it will play the music. By default it plays the FM version of the music, courtesy of the OPL emulator I ripped out of the DosBox source. Given a -M argument, it will play the basic Midi version of the file. If you give it a -C option instead, it will play the Canvas version, with the slightly different instruments. Finally, if you have it playing Midi (with either -M or -C) and supply a final file name, it will use that filename to write the midi stream to before it actually plays the music. For example, to play just the FM music for the tavern: xeenmusic TAVERN.M To hear the Canvas Midi music: xeenmusic TAVERN.M -C To hear the Basic Midi music and output a file: xeenmusic TAVERN.M -M tavern.mid And so on. It currently plays it through whatever is the first midi device on your computer, usually the default software synth that comes with Windows I think. This should be fine for most users, but some users may have different midi devices they want to use: a different software synth they've installed (timidity for example), or if they actually have a sound card with an actual OPL3 chip (do they even still make those?) or a passthrough device if they want to play it to their Midi enabled keyboard. In those cases, I'll have to add some kind of selector. To extract files from the CC data file, try extract.exe. It takes as arguments a CC file, and then the file you want to extract. Easy, no? Use the Filenames table to figure out which files you want and where they are. Hopefully this all makes sense. Enjoy the music! And let me know if you encounter any problems. --WizardStan 21:14, 8 July 2009 (UTC) What do all the "Uh oh, interval is getting too small!" messages mean? There are a couple of issues with the Midi file that gets written to disk with the -C option. The tempo of the Midi file is too fast (although oddly, xeenmusic.exe seems to play the Midi at the right tempo). Also, the volumes on the channels don't seem to be right (even when playing back in xeenmusic.exe). For instance, on MM5's SF05.M, the synth organ(?) should be quieter than the bass guitar, but isn't. As an aside, the -M version seems to match up to the MT-32 version. It sounds a lot better if you have an MT-32 emulator. Finally, out of curiosity, what did you have to do to convert the M note values to Midi note values? -- 11:35, 9 July 2009 (UTC) The notes are supposed to play ever 13.x something milliseconds. SDL doesn't guarantee that level of precision, millisecond like accuracy. You may not notice the difference between a few notes off by 3 millisecond or so, but over the course of a song, you'll notice that it is playing faster or slower. I wrote the timer code to do its best, pausing for as long as it needs to make the whole song flow: For example, the first note it is supposed to pause for 13 ms, but SDL only pauses for 10ms. After the second note, it sends a pause for 16 ms (the 10 ms it really did pause, plus 16 = 26 = 13+13), for which SDL actually pauses for 20 ms. The third time through, it sends a pause for 9ms, for which SDL pauses for 10 ms again. And so on and so forth. Due to the imperfections of the SDL timer, sometimes the delay drops to 0. That triggers the "interval getting too small" warning. It then plays the next note immediately without pause, giving an extra 10 ms to play with. Like I said, from one note to the next, the differences are so small that you don't notice, but over the course of the entire song, you'd notice that it wasn't quite right. At least I did. If your ears can detect the rare moments where the timer skips, then you are a super hero. ;) In the final product, all these warnings will be removed and you'll never know. Or I'll change to a different timer which guarantees better accuracy. Unlike in the EXE where I have a timer I use to wait for the next note, the MIDI file format uses a "beats per second" and "message delta" time stamping system to know when to play the next note. 40 BPS seemed about right, though I guess it is a little fast. Should probably be closer to 30. I'll experiment some more when I get the chance. If you want, you can open a MIDI file up in a hex editor, change offset 13 from 0x28 to 0x1E or something to change the tempo from 40 to 30. I'll take a look at the volume stuff. I may be sending the wrong messages to the synth for volume control. I know it doesn't work in FM, but I thought I got it working in MIDI. Here's the code for converting the .M note data into a MIDI note value: Uint8 midiNotes24 = { 0x00, 0x0C, 0x0E, 0x10, 0x11, 0x13, 0x15, 0x17, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x0B, 0x0D, 0x0F, 0x10, 0x12, 0x14, 0x16}; Uint8 noteToMidi(Uint8 note) { if ((note & 0x1F) > 23) { // A properly formatted music file should never do this!!! std::cerr << "Encountered note " << (int)note << ". note&0x1F = " << (int)(note & 0x1F) << std::endl; } Uint8 freq = midiNotes& 0x1F; freq += (note & 0xE0) >> 2; freq += (note & 0xE0) >> 3; return freq; } --WizardStan 12:33, 9 July 2009 (UTC)